Entity Resolution in a Big Data Framework

Resource Description Framework (RDF)1 is a data model that can be used to publish semistructured data visualized as directed graphs. An example is Dataset 1 in Fig. 1. Nodes in the graph represent entities and edges represent properties connecting these entities. Two nodes may refer to the same logical entity, despite being syntactically disparate. For example, the entity Mickey Beats in Dataset 1 is represented by two syntactically different nodes. Entity Resolution (ER) is the problem of resolving such semantically equivalent entities by linking them using a special sameAs property edge (Ferraram, Nikolov, and Scharffe 2013). The ER problem is not restricted to the RDF data model but can be stated abstractly as identifying and resolving semantically equivalent entities in one or more datasets (Elmagarmid, Ipeirotis, and Verykios 2007). As an example of applying ER on relational datasets, consider Datasets 2 and 3 in Fig. 1. In these datasets, an entity is represented as a tuple. The goal is to identify duplicate tuples, that is, tuples referring to the same logical entity. ER is an important AI problem that has been acknowledged as occurring in structured, semistructured and even unstructured data models. A survey on the subject cites at least eight different names for the problem, including record linkage, instance matching, link discovery and co-reference resolution (Elmagarmid, Ipeirotis, and Verykios 2007). The problem has grown in concert with the Semantic Web and with the publishing of new data on the Web. Given the prevalence of large datasets in data integration applications (Goodhue, Wybo, and Kirsch 1992), an ER solution must be scalable. For example, consider Linked Open Data (LOD2), which is the collection of RDF datasets published under an open license (Bizer, Heath, and Berners-Lee 2009). LOD currently contains over 30 billion triples and over 500 million property edges, published in over 300 datasets. Studies have suggested that LOD contains many syntactically disparate but semantically equivalent entities that have not yet been discovered and linked (Papadakis et al. 2010). In the relational domain, the Deep Web, which is the collection of back-end relational databases powering Web queries and faceted search, has also shown super-linear growth and is at